“If one person builds it, everyone will die”: Why Eliezer Yudkowsky fears overpowered AI

There are people who believe that artificial intelligence (AI) will cure all ills—or at least the ailing economy. There are people who find AI scary. And there's Eliezer Yudkowsky. He's waiting for the end of the world. His new book, "If One Builds It, Everyone Die—Why Superhuman AI Would Kill Us All," was published on September 16.

For a quarter of a century, Yudkowsky has been pondering futuristic scenarios. He conjures up the doom of humanity in blog posts and hour-long tech podcasts . Yudkowsky isn't charming. He comes across as complacent and becomes impatient when his counterpart refuses to accept the superiority of his arguments.

Yudkowsky isn't a programmer either. He never attended high school or university. Yet influential people listen to him – beyond Silicon Valley. Former Federal Reserve Chairman Ben Bernanke recommends his new book. Emmett Shear, interim CEO of Open AI, comments: "It explains clearly and unequivocally why current AI development is extremely dangerous."

How is it that a nerdy teenager who couldn't wait for the sci-fi future to arrive has become the loudest voice when it comes to the extinction of humanity by artificial intelligence? And how convincing are his arguments really?

Born in 1979 into an Orthodox Jewish family in Chicago, Eliezer Yudkowsky realized early on that he was different. At 11, he argued with his parents, a psychiatrist and a physicist, because they clung to their faith despite his better arguments. They said that experience was also important, not just logic. The 11-year-old concluded that his parents were obviously too stupid to understand his arguments.

He recounts this years later in a blog post , in which he seeks other reasons for his parents' shortsightedness. Looking back, it seems implausible to him that his parents were less intelligent than a prepubescent child. Even though he, Eliezer, was supposedly exceptionally smart. By his own account, he scored as high as the very best college applicants on the American SAT at age 11.

Yudkowsky didn't last long at school. From seventh grade onward, he taught himself what he deemed necessary at home. He was particularly interested in science fiction. At 17, Yudkowsky declared his sole purpose in life to advance the "singularity." This is the term used by tech prophets for a hypothetical point in time at which AI surpasses human intelligence and develops exponentially on its own.

Yudkowsky met like-minded people online. He landed on the mailing list of the "Extropians," an obscure group that included the future WikiLeaks founder Julian Assange and futurist Ray Kurzweil. Two Extropians, the entrepreneurial couple Brian and Sabine Atkins, were so enthusiastic about Yudkowsky's contributions that they paid the then 21-year-old a monthly salary to ponder the rosy future of superintelligence. Thus was born the " Singularity Institute for Artificial Intelligence ," a private research group that Yudkowsky still leads today.

The institute now has a different name. After just a few months, Yudkowsky had come to a shocking realization: Once artificially intelligent beings were created, they could wipe out humanity. From then on, his priority was to research how to train AI to act ethically. The funders agreed with the change of course.

During this time, Yudkowsky met Peter Thiel – co-founder of PayPal and early investor in Facebook, one of the most successful and influential entrepreneurs in Silicon Valley. Thiel co-financed Yudkowsky's institute and helped him network, for example through the Singularity conferences, which Yudkowsky now organized at Stanford and to which prominent speakers were invited. At one of these, he introduced young entrepreneur Demis Hassabis to Thiel – thus contributing to the success of one of the best AI startups: Hassabis founded Deepmind, which is now part of Google.

Yudkowsky writes that he now regrets having contributed to Deepmind's success and thus to the acceleration of AI development. Conversely, Peter Thiel now considers him a lost Luddite.

In fact, Yudkowsky's warnings about malevolent superintelligence have become increasingly dire over the past decade. He hasn't come up with any idea how to train AI to be good. He even believes it's almost impossible.

The new book elaborates on the argument. Yudkowsky wrote it with the help of Nate Soares, the president of his research institute . It goes like this:

1. AI is not built, but bred. Today's AI is not programmed according to fixed rules, but rather evolves through learning algorithms . Therefore, its functional mechanisms are opaque and difficult to predict.
2. A superintelligence will not behave as expected . Here, the authors argue with evolution. Humans are programmed for eating and reproduction. Yet they have invented sweeteners and condoms, which contradict evolutionary purpose. Proof that even with an AI given clear goals, unexpected behavior can emerge.
3. Humanity has only one attempt to keep the super AI in check: Some researchers are developing methods to control the AI. Yudkowsky believes this is unlikely to work: Because until the super AI is there, one cannot test its controls. And if it is there and they don't work, then humanity is out of luck.
4. There's nothing to suggest that an alien superintelligence would leave us alone. Here, too, evolution serves as a case in point: Humans have subordinated all less intelligent beings in the world to their own purposes. There's no compelling reason why AI wouldn't do the same. Especially since humans would likely disrupt artificially intelligent beings or compete with them for resources.

Yudkowsky argues rigorously—albeit within a very specific worldview. He addresses all those who are convinced that in the foreseeable future there will be superior, intelligent machines that are autonomous and as alien to humans as insects or aliens. He argues convincingly why such artificial beings have little reason to subordinate themselves to humans.

The major weakness of his book is that he assumes that humanity is close to building superintelligence, but he doesn't provide any good arguments for this.

Yudkowsky assumes that superintelligence will be built at some point—simply because it's theoretically possible. He draws arguments for his theory from history. Shortly before the invention of the airplane, it was also considered science fiction.

In his view, there's no need for further evidence that superintelligence could soon arrive. Because it would have catastrophic consequences, it should be prevented as a precautionary measure, regardless of how likely its occurrence is.

The authors are prepared to resort to drastic measures. "If everyone believed even a quarter of what we believe, (. . .) they would take to the streets and shut down data centers." They advocate regulating AI research through international treaties, like nuclear power. And anyone who doesn't comply might have their data centers bombed.

The tone of the book is almost pleading. This is probably because the authors know that the best moment for their arguments has passed for now. In 2023, there was a window of attention. At that time, Chat-GPT, the bot that lets you talk like a human, had just been released. Scenarios from science fiction movies suddenly seemed realistic to many.

Entrepreneurs and politicians warned of the extinction of humanity through AI. Yudkowsky gave a TED Talk on the question: "Will superintelligent AI destroy the world?" and published a guest article in Time magazine, which named him one of the 100 most influential voices in AI.

Today, the panic of that time seems exaggerated. AI has become part of everyday life – and although it has amazing capabilities and comes with risks , it is now clear: chatbots are a technology, not an alien species with an inner life and aspirations to power.

The main problem with Yudkowsky's book is that it obscures this: On the one hand, the authors vividly explain how today's AI works. But then they describe its capabilities as obscure and uncontrollable, implicitly attributing to it an inner life it doesn't have. It seems like a device to make the problem of murderous superintelligence seem more urgent than it actually is.

That doesn't mean his warnings are worthless. There's one group that should take them more seriously: those tech entrepreneurs who claim they're developing superintelligent, godlike beings. Yudkowsky's explanation of why that would be a bad idea is convincing.

Eliezer Yudkowsky and Nate Soares: If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. Little, Brown and Company, Boston 2025. 272 ​​pp., $27.90.